Enhanced data storage and transport via wavefront multiplexing

ABSTRACT

For data writing, a first wavefront multiplexing (WFM) processor performs WFM on M input streams to generate N output streams. A pre-processor segments or codes a source stream to produce the M input streams. For data reading, a first wavefront demultiplexing (WFD) processor performs WFD on M input streams to generate N output streams. A post-processor de-segments or decodes the N output streams into a source stream.

RELATED APPLICATIONS

This application claims priority from Provisional Patent Application No.62/311,816, filed on Mar. 22, 2016. This application is related to U.S.Pat. No. 8,098,612 issued on Jan. 17, 2012, entitled “APPARATUS ANDMETHOD FOR REMOTE BEAM FORMING FOR SATELLITE BROADCASTING SYSTEMS”; U.S.Pat. No. 8,111,646 issued on Feb. 7, 2012, entitled “COMMUNICATIONSYSTEM FOR DYNAMICALLY COMBINING POWER FROM A PLURALITY OF PROPAGATIONCHANNELS IN ORDER TO IMPROVE POWER LEVELS OF TRANSMITTED SIGNALS WITHOUTAFFECTING RECEIVER AND PROPAGATION SEGMENTS”; U.S. patent applicationSer. No. 14/712,145, filed on May 14, 2015, entitled “SURVIVABLE CLOUDDATA STORAGE AND TRANSPORT”; and U.S. patent application Ser. No.14/512,959, filed on Oct. 13, 2014, entitled “ENVELOPING FOR CLOUDCOMPUTING VIA WAVEFRONT MUXING”, which are expressly incorporated byreference herein in their entireties.

TECHNICAL FIELD

One disclosed aspect of the embodiments is directed to the field of datastorage and transport. In particular, the embodiment is directed to datastorage and transport using wavefront multiplexing (WFM) technology.

BACKGROUND

Long before the beginning or digital age, people had manually storeddata while the ‘data storage’ from time to time might suffer loss due tolack of availability and privacy protection. With the advancement ofdigital technology, data storage has been an indispensable function inmany aspects of modern era. The need for availability and privacyprotection remains central to evolving data storage design.

Data not only resides in storage but also appears in transition amongcommunication terminals and users. To provide quality of service andquality of experience, it is also of significant value to transport datathat is highly available and securely protected. The service of datatransport should meet requirements of availability and privacyprotection to satisfy user's demand for quality and experience.

Repetition coding is one approach to providing availability against theevent of data loss. One application of repetition code is RAID(redundant array of independent disks). Among variations of RAID, RAID 1creates one redundant piece of a data stream. For one data stream, RAIDthus creates two identical copies to be stored. The space overhead ofRAID 1 is 50%, which is high in state-of-the-art storage, and it bearslow level privacy protection if no encoding or other measure is furtherapplied to the stored copy.

Wavefront multiplexing (WF muxing, or K-muxing) and wavefrontdemultiplexing (WF demuxing or K-demuxing) are multi-dimension dataprocessing methods. Both K-muxing and K-demuxing define transformationof multi-dimensional signals or data streams that feature particulardistribution patterns (or ‘wavefronts’) in K-space. K-muxing andK-demuxing enable redundancy to enhance availability and providescrambled signals or data streams designed toward privacy protection.

SUMMARY

One disclosed aspect of the embodiments is a method and apparatus toprovide data storage and transport using wavefront multiplexing (WFM)technique. For data writing, a first wavefront multiplexing (WFM)processor performs WFM on M input streams to generate N output streams.A pre-processor segments or codes a source stream to produce the M inputstreams. For data reading, a first wavefront demultiplexing (WFD)processor performs WFD on M input streams to generate N output streams.A post-processor de-segments or decodes the N output streams into asource stream.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments may best be understood by referring to the followingdescription and accompanying drawings that are used to illustrateembodiments. In the drawings:

FIG. 1 is a diagram illustrating a system using a data transport and/orstorage processing system according to one embodiment.

FIG. 2 is a diagram illustrating a data transport and/or storageprocessing system for transmitting or writing data to a storage systemaccording to one embodiment.

FIG. 3 is a diagram illustrating an architecture for the data transportand/or storage processing system according to one embodiment.

FIG. 4 is a diagram illustrating a data transport and/or storageprocessing system for transmitting or writing data to three localstorage systems and one cloud storage device according to oneembodiment.

FIG. 5 is a diagram illustrating a data transport and/or storageprocessing system for transmitting or writing data to two local storagesystems and two cloud storage devices according to one embodiment.

FIG. 6 is a diagram illustrating a data transport and/or storageprocessing system for transmitting or writing data to a storage systemhaving two devices and two cloud storage devices according to oneembodiment.

FIG. 7 is a diagram illustrating a data transport and/or storageprocessing system for transmitting or writing data to a storage systemand four cloud storage devices according to one embodiment.

FIG. 8 is a diagram illustrating a data transport and/or storageprocessing system for transmitting or writing data using a systematiccoder according to one embodiment.

FIG. 9 is a diagram illustrating a data transport and/or storageprocessing system for transmitting or writing data using a cascadedstructure for the WFM processor according to one embodiment.

FIG. 10 is a diagram illustrating a data transport and/or storageprocessing system for transmitting or writing data using a cascadedstructure for the WFM processor according to one embodiment.

FIG. 11 is a diagram illustrating a data transport and/or storageprocessing system for receiving or reading data from a storage systemaccording to one embodiment.

FIG. 12 is a diagram illustrating a data transport and/or storageprocessing system for receiving or reading data from a storage systemand a cloud storage according to one embodiment.

FIG. 13 is a diagram illustrating a data transport and/or storageprocessing system for receiving or reading data from a storage systemand a cloud storage according to one embodiment.

FIG. 14 is a diagram illustrating a data transport and/or storageprocessing system for receiving or reading data using a systematicdecoder according to one embodiment.

FIG. 15 is a diagram illustrating a data transport and/or storageprocessing system for receiving or reading data using a cascadedstructure for the WFD processor according to one embodiment.

FIG. 16 is a diagram illustrating a data transport and/or storageprocessing system for receiving or reading data using a cascadedstructure for the WFD processor according to one embodiment.

FIG. 17 is a diagram illustrating curves representing failure rate of adistribute storage system according to one embodiment.

FIG. 18 is a diagram illustrating curves representing failure rate of adistribute storage system according to one embodiment.

FIG. 19 is a diagram illustrating a WF processor according to oneembodiment.

DETAILED DESCRIPTION

One disclosed aspect of the embodiments is a method and apparatus toprovide data storage and transport using wavefront multiplexing (WFM)technique. The technique allows writing data to or reading data fromstorage devices in a distributed manner to enhance fault tolerance,reliability, and availability.

For data writing, a first wavefront multiplexing (WFM) processorperforms WFM on M input streams to generate N output streams. Apre-processor segments or codes a source stream to produce the M inputstreams. The N output streams are stored in at least one of a pluralityof storage devices. For cascade operation, a second WFM processorperforms WFM on the N output streams to produce storage streams to bestored in at least one of a plurality of storage devices. The pluralityof storage devices includes at least one of a network attached storage(NAS) device, a direct access storage (DAS) device, a storage areanetwork (SAN) device, a redundant array of independent disks (RAIDs), acloud storage device, a hard disk, a solid-state memory device, and adevice capable of storing data.

For data reading, a first wavefront demultiplexing (WFD) processorperforms WFD on M input streams to generate N output streams. Apost-processor de-segments or decodes the N output streams into a sourcestream. The M input streams are retrieved from at least one of aplurality of storage devices. For cascade operation, a second WFDprocessor performs WFD on K storage streams from at least one of aplurality of storage devices to produce the M input streams. Theplurality of storage devices includes at least one of a NAS device, aDAS device, a SAN device, RAIDs, a cloud storage, a hard disk, asolid-state memory device, and a device capable of storing data.

In the following description, numerous specific details are set forth.However, it is understood that embodiments may be practiced withoutthese specific details. In other instances, well-known circuits,structures, and techniques have not been shown to avoid obscuring theunderstanding of this description. One disclosed feature of theembodiments may be described as a process which is usually depicted as aflowchart, a flow diagram, a structure diagram, or a block diagram. Oneembodiment may be described by a schematic drawing depicting a physicalstructure. It is understood that the schematic drawing illustrates thebasic concept and may not be scaled or depict the structure in exactproportions.

The term “writing” refers to the act of storing data on or transmittingor sending data through multiple physical and logical dimensions. Theterm “reading” refers to the act of retrieving data from or receivingdata through multiple physical and logical dimensions. Physicaldimensions may refer to computers, mobile devices, data centers and soon. Logical dimensions may refer to allocated or virtualized resourcesfor data storage or data transport. Both physical and logical dimensionsmay also refer to communication channels in general.

One disclosed aspect of the embodiments relates to distributed datastorages with built-in redundancy for a single stream data subdividedinto multiple (M) data substreams or M independent data streams,converted into K-muxed domain with M+N output wavefront components(WFCs), and stored these M+N WFC output data as M+N separated datastorage sets, where N, and M are non-negative integers. As a result, thestored data sets are WFCs in the format of linear combinations of thedata sets, instead of the data sets themselves. The coefficientsinvolved in K-muxing and K-demuxing may take complex values. Hence thevector of coefficients involved in K-muxing and K-demuxing may include,but not limited to, column vectors in Hadamard transformation, Fouriertransformation, etc. The matrix comprising coefficients involved inK-muxing and K-demuxing features subsets of M rows that have full rankin order to satisfy the redundancy requirements.

In general, the input ports of a K-muxing transform are referred to as“slices” and the output ports are referred to as “WFCs”. For instance,the first and the third input ports to a 16-to-16 K-muxing transform arereferred as the slice 1 and the slice 3, respectively. Similarly the13^(th) and the 16^(th) output ports are called the WFC 13 and theWFC16, respectively. Collectively, the output data from a K-muxingtransform also referred as the K-muxed data are outputted from all theWFC ports. A first input stream connected to slice 1 of the 16-to-16 Kmuxing transform shall appear in all the WFC ports with a uniquewavefront called wavefront 1 indicated as wavefront vector 1 or WFV1over a 16-dimensional space; each dimension representing an output froma unique WFC port. Similarly a second input stream connected to slice 16of the 16-to-16 K muxing transform shall also appear in all the WFCports with another unique wavefront called wavefront 16 indicated aswavefront vector 16 or WFV16.

Existing redundancy-generation coding such as erasure code often appearsas systematic code, which preserves original data streams in addition tocomputed parity data streams. The preserved original data streams shouldbe protected, unless otherwise further processed by measures such asencryption. On the other hand, K-muxing renders each WFC unintelligibleto protect every data stream to be stored or transported.

Assume, in a writing process, a data stream's M substreams (S₁, S₂, . .. , S_(M)) are transformed to M+N WFCs (D₁, D₂, . . . , D_(M+N)) viaK-muxing. Each WFC D_(i) can be further coded by a coding function thatgenerates coded components (CCs) R_(i,1), R_(i,2), . . . , R_(i,L) to bestored in or transported through multiple physical and logicaldimensions. To ‘read’ the substreams (S₁, S₂, . . . , S_(M)), the set ofCCs {R_(i,1), R_(i,2), . . . , R_(i,L)} (or its subset) associated withD_(i) can be used to first decode D_(i) via a decoding function; andthen a subset (with size no less than M) of the WFCs {D₁, D₂, . . . ,D_(M+N)} can be used to reconstitute S₁, S₂, . . . , S_(M) viaK-demuxing followed by the recovery of the original data stream. Hence,in the writing process, K-muxing can be performed, proceeding theexecution of the coding function. In the corresponding reading process,decoding takes place first, followed by K-demuxing.

Assume, in a writing process, a data stream is transformed by a K-muxer,generating WFCs D₁, D₂, . . . , D_(M+N). A coding function can beenabled to take all WFCs (D₁, D₂, . . . , D_(M+N)) as input, generatingCCs (R₁, R₂, . . . , R_(L)), where L is an integer, as output to bestored in or transported through multiple physical and logicaldimensions. In the corresponding reading process, a decoding functioncan be enabled to take the set of CCs {R₁, R₂, . . . , R_(L)} or itssubset as input, recovering the set of WFCs {D₁, D₂, . . . , D_(M+N)} orits subset as output. A K-demuxer can then be enabled to take the set ofWFCs {D₁, D₂, . . . , D_(M+N)} or its subset as input and thenreconstitute the original data stream.

One can also arrange the K-muxer and coding function as follows. Assume,in a writing process, a data stream is transformed by a K-muxer,generating WFCs D₁, D₂, . . . , D_(M+N). Several coding functions can beenabled in parallel, each of which takes one subset of the set {D₁, D₂,. . . , D_(M+N)} as input denoted by {D_(i,1), D_(i,2), . . . ,D_(i,Q)}, where Q is an integer, and generates a set of CCs {R_(i,1),R_(i,2), . . . , R_(i,L)} to be stored in and transported throughmultiple physical and logical dimensions. In the corresponding readingprocess, all or some decoding functions can be enabled, each of whichcan take one subset of some CC set {R_(i,1), R_(i,2), . . . , R_(i,L)}as input and generate a set of WFCs {D_(i,1), D_(i,2), . . . , D_(i,Q)}or its subset as output. A K-demuxer can then be enabled to take the setof WFCs {D₁, D₂, . . . , D_(M+N)} or its subset (with size no less thanM) as input and then reconstitute the original data stream.

The K-muxer and coding function can also be arranged in differentorders. Assume, in a writing process, a data stream is encoded by acoding function, generating CCs R₁, R₂, . . . , R_(M). A K-muxer can beenabled to take all CCs (R₁, R₂, . . . , R_(M)) as input, generating M+NWFCs (D₁, D₂, . . . , D_(M+N)) as output to be stored in or transportedthrough multiple physical and logical dimensions. In the correspondingreading process, a K-demuxer can be enabled to take a subset (with sizeno less than M) of the WFCs (D₁, D₂, . . . , D_(M+N)) as input,generating the set of CCs {R₁, R₂, . . . , R_(M)} or its subset asoutput. A decoding function can then be enabled to take the set of CCs{R₁, R₂, . . . , R_(M)} or its subset as input and then reconstitute theoriginal data stream.

One can also arrange the K-muxer and coding function as follows. Assume,in a writing process, a data stream is encoded by a coding function,generating CCs R₁, R₂, . . . , R_(L). Several K-muxers can be enabled inparallel, each of which takes one subset of the set {R₁, R₂, . . . ,R_(L)} as input denoted by {R_(i,1), R_(i,2), . . . , R^(i,M)} andgenerates a set of WFCs {D_(i,1), D_(i,2), . . . , D_(i,(M+N))} to bestored in and transported through multiple physical and logicaldimensions. In the corresponding reading process, all or some K-demuxerscan be enabled, each of which can take one subset (with size no lessthan M) of some WFC set {D_(i,1), D_(i,2), . . . , D_(i,(M+N))} as inputand generate a set of CCs {R_(i,1), R_(i,2), . . . , R_(i,M)} or itssubset as output. A decoding function can then be enabled to take theset of CCs {R₁, R₂, . . . , R_(M)} or its subset as input and thenreconstitute the original data stream.

K-muxers and K-demuxers can also be cascaded in designated orderaccording to the requirements of resource allocation, as disclosed inthis disclosure.

FIG. 1 is a diagram illustrating a system 100 using a data transportand/or storage processing system according to one embodiment. The system100 includes a data transport and/or storage processing system 110, asource network 120, a source storage system 130, a source computersystem 140, a destination network 170, a destination storage system 180,and a destination computer system 190. Note that the source device maybe the same as the destination device. For example, the source network120 may be the same as the destination network 170. The system 100 maycontain more or less than the above components. The system 100 mayfunction to transport data and write or transmit data to a storagesystem, such as the destination storage system 180. The system 100 mayalso function to transport data and read or receive data from a storagesystem, such as the source storage system 130. In addition, the system100 may function to read or receive data from one end and to write ortransmit data to another end, including both source devices anddestination devices.

The data transport and/or storage processing system may receive or reada stream of data from the source network 120, the source storage system130, or the source computer system 140. The data or stream of data maybe an original stream of data or content that has not been processed bythe processing system 110, or it may have already been processed by theprocessing system 110 and is now ready to be reconstituted to producethe original data or stream of data.

The source network 120 may be any type of network, wired or wireless,including broadband, local area network (LAN), the Internet, intranet,or cloud. The network 120 may connect to any device that have storagecapability or produce content that may be transmitted. In oneembodiment, the network 120 may be connected to storage devices 122 and124. The storage devices 122 and 124 may be any one of a networkattached storage (NAS) device, a direct access storage (DAS) device, ora storage area network (SAN) device. The NAS device may use any suitabledata transmission methods, such as Transmission ControlProtocol/Internet Protocol (TCP/IP), Ethernet. The DAS device may employany of the interfaces such as small computer system interface (SCSI),serial attached SCSI (SAS), Advanced Technology Attachment (ATA), etc.The SAN device may use any suitable interface for data transmission suchas Fiber Channel, IP.

The source storage system 130 may be a highly reliable storage systemsuch as a group of redundant array of independent disks (RAIDs) 130 ₁, .. . , 130 _(M). The RAIDs 130 may be any type of RAIDs that provide dataredundancy, fault tolerance, or performance improvement. Any suitablelevel may be configured. For example, RAID 0 provides striping thatdistributes contents of files among the disks, RAID 1 provides datamirroring in which data is written identically to two drives, therebyproducing a “mirrored set” of drives.

The source computer system 140 may be any suitable computer systemhaving storage capability, including a server, a desktop computer 142, alaptop computer, a mobile device such as panel computer or telephone,video or image capture device, etc. It may include storage devices suchas hard disk 144, solid-state drive 146, or thumb drive 148.

The data from the source network 120, the source RAIDs 130, or thesource computer system 140 are transferred to the processing system 110via a bus or channel 150.

The processing system 110 processes the data and transmits, sends,writes, or stores the processed data to a destination device, includingthe destination network 170, the destination storage device 180, and thedestination computer system 190. Similar to their source counterparts,the destination network 170 may connect to storage devices 172 and 174.The storage devices 172 and 174 may be any one of a NAS device, a DASdevice, or a SAN device. The destination storage device 180 may haveRAIDs 180 ₁, . . . , 180 _(N); and the destination computer system 190may have a desktop computer 192, a hard drive 194, a solid-state drive(flash devices) 196, and a thumb drive 198. The writing or storing datainto these destination devices may be performed in a distributed manner.In other words, output data streams from the processing system 110 maybe distributed over any combination of these destination devices. Forexample, if there are 4 output streams from the processing system 110,three may be stored in the RAIDs 180, and one may be stored in a cloudstorage device.

The system 100 may operate in a writing mode or a reading mode. In thewriting mode, a source stream S is available to be processed and writtenor stored in any of the destination devices 170/180/190. There are anumber of embodiments in the writing mode, shown in FIGS. 2, 4-10. Inthe reading mode, a number of storage streams are available from a leasta storage device 120/130/140 to be processed to recover or reconstitutethe source stream S. There are a number of embodiments in the readingmode, shown in FIGS. 11-16. In essence, the process in the reading modeof the data streams D_(i)'s operates in reverse of the process thatwrites the data streams D_(i)'s to the storage device(s).

FIG. 2 is a diagram illustrating the data transport and/or storageprocessing system 110 for transmitting or writing data to a storagesystem according to one embodiment. The processing system 110 mayinclude a segmenter 210 and a WFM processor 220. The processing system110 may include more or less than the above components. For clarity,components of the storage system 170/180/190 are shown in FIG. 2 as RAID1 232, 234, 236, and 238. In other embodiments, any of the storagedevices 170/180/190 may be used.

The segmenter 210 is a pre-processor that pre-processes the sourcestream S, which comes from a source device (e.g., the source network120, the source storage system 130, or the source computer system 140)to produce the M input streams. In the illustrative example shown inFIG. 2, M=3. In other words, the segmenter 210 splits the source streamS into 3 data streams or segments S₁, S₂, and S₃. The splitting may beperformed using a pre-determined method such as permutation.

The WFM processor 220 performs WFM on the M input streams to generate Noutput streams as the WF components (WFC). In the illustrative examplein FIG. 2, M=3 and N=4. So, the WFM processor 220 performs the WFM onthe 3 input streams or segments S₁, S₂, and S₃ to generate 4 outputstreams D₁, D₂, D₃, and D₄. The WFM is essentially a matrixmultiplication of the input vector S=(S₁, S₂, S₃)^(T) (T indicates atranspose vector) and the coefficient matrix [w_(ij)] as follows:

$\begin{matrix}{\begin{bmatrix}D_{1} \\D_{2} \\D_{3} \\D_{4}\end{bmatrix} = {{\begin{bmatrix}w_{11} & w_{12} & w_{13} \\w_{21} & w_{22} & w_{23} \\w_{31} & w_{32} & w_{33} \\w_{41} & w_{42} & w_{43}\end{bmatrix}\begin{bmatrix}S_{1} \\S_{2} \\S_{3}\end{bmatrix}}.}} & (1)\end{matrix}$

Equation (1) gives rise to the following:D ₁ =w ₁₁ S ₁ +w ₁₂ S ₂ +w ₁₃ S ₃  (2a)D ₂ =w ₂₁ S ₁ +w ₂₂ S ₂ +w ₂₃ S ₃  (2b)D ₃ =w ₃₁ S ₁ +w ₃₂ S ₂ +w ₃₃ S ₃  (2c)D ₄ =w ₄₁ S ₁ +w ₄₂ S ₂ +w ₄₃ S ₃  (2d)

As seen from the above equations, each of the output streams D_(i)'s(i=1, 2, 3, 4), may be considered as a linear combination of thecoefficients w_(ij)'s (i=1, 2, 3, 4; j=1, 2, 3), and the input streamsS_(j)'s (j=1, 2, 3). To solve for S_(j)'s (j=1, 2, 3), we need onlythree independent equations. Since there are 4 equations, one isextraneous and may be ignored. For example, the output D₄ may not beused. Alternatively, all 4 may be used with one is redundant, used forincreasing fault tolerance in case one of the three outputs is in erroror lost. Suppose D₄ is not used, the above set of equations reduces to(2a), (2b) and (2c) which can be solved by a number of methods such assubstitution, elimination, or Kramer's rule, as are well known by oneskilled in the art.

The three column vectors of the matrix in (1) represent three‘wavefronts’ that feature three distribution patterns of segments S₁, S₂and S₃ respectively. Each coefficient w_(ij) can take real or complexvalue. As discussed above, any sub-matrix comprising three rows of thematrix in (1) has full rank in order to fulfill the redundancyrequirements: any three wavefront components (WFCs) of D₁, D₂, D₃ and D₄are sufficient to recover three segments S₁, S₂ and S₃.

Another way to envision this transformation is to assume there are 4input streams S₁, S₂, S₃, and S₄, and the input vector [S] is a columnvector with 4 components where S₄ is set to zero. The coefficient matrixtherefore may be organized as a 4×4 matrix. The matrix multiplicationmay be performed as follows:

$\begin{matrix}{\begin{bmatrix}D_{1} \\D_{2} \\D_{3} \\D_{4}\end{bmatrix} = {{\begin{bmatrix}w_{11} & w_{12} & w_{13} & w_{14} \\w_{21} & w_{22} & w_{23} & w_{24} \\w_{31} & w_{32} & w_{33} & w_{34} \\w_{41} & w_{42} & w_{43} & w_{44}\end{bmatrix}\begin{bmatrix}S_{1} \\S_{2} \\S_{3} \\0\end{bmatrix}}.}} & (3)\end{matrix}$

The output from each WFC is processed by RAID 1 that performs mirroring,namely replication. Data storage sites or devices 232, 234, 236, and 238perform ‘mirroring’ functions such that D_(i)=R_(i,1)=R_(i,2), i=1, 2,3, 4. Four sets {R_(i,1), R_(i,2)}, i=1, 2, 3, 4, may be stored in fourphysical and logical dimensions such as four separate network-attachedstorage (NAS) sites or devices. These NAS sites may be local NAS sites,on private cloud or on public cloud. One such distribution may featurethree local NAS sites and the remaining one in a storage site on publiccloud. The local distribution of three WFM data sites will be sufficientfor reconstituting the stored data, while the one on cloud providesadditional redundancy.

The WFM processor 220 may also be re-configured to take a known datastream as a 4^(th) input (not shown). This ‘injected’ data stream mayappear as a dominating ‘envelope’ over the four WFCs D₁, D₂, D₃ and D₄.Systems, methods and apparatus for digital enveloping have beendiscussed extensively in the U.S. patent application Ser. No.14/512,959, filed on Oct. 13, 2014. The WFM processor 220 may performWFM on the M input streams including an envelope to generate the Noutput streams including an enveloped output stream which issubstantially identical to the envelope.

FIG. 3 is a diagram illustrating an architecture for the data transportand/or storage processing system 220 according to one embodiment. Thearchitecture correspond to the 4×4 matrix shown in Equation (2) above.The processing system 220 includes a storage device 310 such as a memorythat stores the coefficients w_(jk)'s (j,k=1, . . . , 4), multipliers322, 324, 326, and 328 and an adder 330. For fully parallel operations,four sets of the 4 multipliers and one adder will be needed. Anycombination of devices may be employed. For example, a single multiplierand a 2-input adder may be used where the multiplier performsmultiplication sequentially and the adder acts like an accumulator toaccumulate the partial products. The input S₄ may be unused or us anenvelope for envelope processing as discussed above. The fourmultipliers 322, 324, 326, and 328 and the adder 330 may form a linearcombiner that perform a linear combination of the coefficients w_(jk)'sand the input streams S_(k)'s as discussed above.

It should also be noted that while the architecture 220 is shown for theWFM processor, it is also applicable for the WFD processor because bothtypes of processor involve a matrix multiplication. The differences arethe types of inputs and outputs and the matrix coefficients in thememory 310.

FIG. 4 is a diagram illustrating the data transport and/or storageprocessing system 110 for transmitting or writing data to three localstorage systems and one cloud storage device according to oneembodiment. The processing system 110 in FIG. 4 is similar to the system110 in FIG. 2 except that the RAID 1 device 238 is replaced by thenetwork cloud 170 and a storage device R₄ 420.

The WFM processor 220 performs WFM on the three input streams S₁, S₂ andS₃ and generates the four output streams WFCs D₁, D₂, D₃ and D₄ as givenin equation (1) above. The three output streams D₁, D₂, D₃ are writtenor stored in three local storage devices 232, 234, and 236, respectively(e.g., local NAS sites). The output stream D₄ may be stored in a publicstorage R₄ 420 via cloud 170. As discussed above, the data storedlocally are sufficient to recover the segmented streams S₁, S₂, and S₃.In case one is lost or the corresponding NAS site fails, the data D₄ maybe retrieved from the cloud storage 420. It then can be used togetherwith the remaining two data streams to recover the segmented streams S₁,S₂, and S₃.

FIG. 5 is a diagram illustrating the data transport and/or storageprocessing system 110 for transmitting or writing data to two localstorage systems and two cloud storage devices according to oneembodiment. The processing system 110 in FIG. 5 is similar to the system110 in FIG. 2 except that the RAID 1 device 238 and RAID 1 device 236are replaced by the network cloud 170 and two storage devices R₃ 520 andR₄ 420.

As discussed above, the two data streams D₁ and D₂ stored in the localNAS devices 232 and 234 are not sufficient to recover the segmentedstreams S₁, S₂, and S₃. One data stream stored on the cloud devices R₃520 and R₄ 420 may be retrieved to be used together with the two datastreams D₁ and D₂ to recover the segmented streams S₁, S₂, and S₃.

FIG. 6 is a diagram illustrating a data transport and/or storageprocessing system for transmitting or writing data to a storage systemhaving two devices and two cloud storage devices according to oneembodiment. The processing system 110 in FIG. 6 is similar to theprocessing system 110 in FIG. 5 except that the two NAS sites RAID 1device 232 and RAID 1 device 234 are replaced by a local NAS site 620that stores D₁ and D₂ in a RAID 1 manner (i.e., mirroring).

As above, the two data streams D₁ and D₂ stored in the local NAS device620 are not sufficient to recover the segmented streams S₁, S₂, and S₃.One data stream stored on the cloud devices R3 520 and R4 420 may beretrieved to be used together with the two data streams D₁ and D₂ torecover the segmented streams S₁, S₂, and S₃.

FIG. 7 is a diagram illustrating the data transport and/or storageprocessing system 110 for transmitting or writing data to a storagesystem and four cloud storage devices according to one embodiment. Theprocessing system 110 is similar to the processing system 110 in FIGS.2, 4-6 except in the destination storage devices. In FIG. 7, the 4output streams D₁, D₂, D₃, and D₄ are stored in local NAS site 720 in aRAID 0 configuration and are also stored in four storage devices R₁ 722,R₂ 724, R₃ 520, and R₄ 420.

In the local NAS site 720, four storage devices store all four but notredundantly. Therefore, while there is no local redundancy, any three ofthe data streams may be retrieved to reconstitute the segmented streamsS₁, S₂, and S₃. If one or two of the devices fail, the data streams maybe retrieved from the corresponding cloud storage devices.

FIG. 8 is a diagram illustrating the data transport and/or storageprocessing system 110 for transmitting or writing data using asystematic coder according to one embodiment. The processing system 110includes a systematic code 810 and the WFM processor 220. The WFMprocessor 220 is similar to the WFM processor 220 in FIG. 2 andtherefore does not need further description. Similarly, the writing orstoring the four output streams D₁, D₂, D₃, and D₄ may be any one of thepreviously described schemes in FIGS. 2-7 and therefore is not describedfurther.

The systematic coder 810 transforms or converts the source stream S intothree input streams S₁, S₂, and S₃. The systematic coder 810 encodes thesource stream S with a systematic code and then splits the encodedstream into three input streams S₁, S₂, and S₃. A systematic code may beany error-correcting code in which the data in the source stream isembedded in the encoded data. For example, checksums and hash functionsmay be combined with the source stream. As another example, S₃ may bethe parity data stream as a numerical combination of S₁ and S₂. Any twoof the three input streams S₁, S₂, and S₃ may be used to reconstitutethe source stream S.

FIG. 9 is a diagram illustrating the data transport and/or storageprocessing system 110 for transmitting or writing data using a cascadedstructure for the WFM processor according to one embodiment. Theprocessing system 110 in FIG. 9 is similar to the processing system 110in FIG. 2 except that the WFM operation is performed by additional WFMprocessors arranged in a serially cascaded configuration.

The cascaded structure includes two levels of WFM processors. In thefirst level, a first WFM processor performs WFM on M input streams togenerate N output streams. In the second level, a second WFM processorperforms WFM on the N output streams to produce storage streams to bestored in a storage device. In the illustrative example in FIG. 9, thefirst level WFM processor is the WFM processor 220 and the second WFMprocessor includes two WFM processors 922 and 924 each operating on asubset of N data streams. Specifically, the WFM processor 220 performsWFM on the input streams S₁, S₂, and S₃ to produce the four outputstreams D₁, D₂, D₃, and D₄, The WFM processor 922 performs WFM on twostreams D₁ and D₂, to generate four storage streams R_(1,1), R_(1,2),R_(1,3), and R_(1,4). The WFM processor 924 performs WFM on two streamsD₃ and D₄, to generate four storage streams R_(2,1), R_(2,2), R_(2,3),and R_(2,4).

The WFM performed by the WFM processor 922 and 924 is similar to thatperformed by the WFM 220 except the number of inputs and the matrixcoefficients are different. The WFM processor 922 performs the WFM as amatrix multiplication as follows:

$\begin{matrix}{\begin{bmatrix}R_{1,1} \\R_{1,2} \\R_{1,3} \\R_{1,4}\end{bmatrix} = {{\begin{bmatrix}\rho_{11} & \rho_{12} \\\rho_{21} & \rho_{22} \\\rho_{31} & \rho_{32} \\\rho_{41} & \rho_{42}\end{bmatrix}\begin{bmatrix}D_{1} \\D_{2}\end{bmatrix}}.}} & (4)\end{matrix}$

Similarly as in FIG. 2, the coefficient ρ_(ij)'s may take real orcomplex values. Any sub-matrix comprising two rows of the matrix in (4)has full rank in order to fulfill the redundancy requirements: any twoWFCs of R_(1,1), R_(1,2), R_(1,3) and R_(1,4) are sufficient to recovertwo WFCs D₁ and D₂. The WFM processor 924 may follow a similarconfiguration: any two WFCs of R_(2,1), R_(2,2), R_(2,3) and R_(2,4) aresufficient to recover two WFCs D₃ and D₄.

The writing or storing of the storage streams R_(1,1), R_(1,2), R_(1,3)and R_(1,4) and R_(2,1), R_(2,2), R_(2,3) and R_(2,4) is similar to theembodiments described earlier in FIGS. 2, 4-6.

FIG. 10 is a diagram illustrating the data transport and/or storageprocessing system 110 for transmitting or writing data using a cascadedstructure for the WFM processor according to one embodiment. Theprocessing system 110 in FIG. 10 is similar to the processing system 110in FIG. 9 except that the WFM processors in the second level eachgenerate three storage streams. The processing system 110 includes thesegmenter 210, the WFM processor 220, and two WFM processors 1022 and1034.

The WFM processor 1022 performs WFM on two streams D₁ and D₂, togenerate three storage streams R_(1,1), R_(1,2), and R_(1,3). The WFMprocessor 924 performs WFM on two streams D₃ and D₄, to generate threestorage streams R_(2,1), R_(2,2), and R_(2,3).

The WFM performed by the WFM processor 1022 and 1024 is similar to thatperformed by the WFM 220 except the number of inputs and the matrixcoefficients are different. The WFM processor 1022 performs the WFM as amatrix multiplication as follows:

$\begin{matrix}{\begin{bmatrix}R_{1,1} \\R_{1,2} \\R_{1,3}\end{bmatrix} = {{\begin{bmatrix}\sigma_{11} & \sigma_{12} \\\sigma_{21} & \sigma_{22} \\\sigma_{31} & \sigma_{32}\end{bmatrix}\begin{bmatrix}D_{1} \\D_{2}\end{bmatrix}}.}} & (5)\end{matrix}$

Similarly as in FIG. 9, the coefficient ρ_(ij)'s may take real orcomplex values. Any sub-matrix comprising two rows of the matrix in (5)has full rank in order to fulfill the redundancy requirements: any twoWFCs of R_(1,1), R_(1,2), and R_(1,3) are sufficient to recover two WFCsD₁ and D₂. The WFM processor 1024 may follow a similar configuration:any two WFCs of R_(2,1), R_(2,2), and R₂₃ are sufficient to recover twoWFCs D₃ and D₄.

The writing or storing of the storage streams R_(1,1), R_(1,2), andR_(1,3) and R_(2,1), R_(2,2), and R_(2,3) is similar to the embodimentsdescribed earlier in FIGS. 2, 4-6.

FIG. 11 is a diagram illustrating the data transport and/or storageprocessing system 110 for receiving or reading data from a storagesystem according to one embodiment. The processing system 110 includesstorage devices 1112, 1114, and 1116, WF de-multiplexing (WFD) processor1120, and a de-segmenter 1130. The processing system 110 may includemore or less than the above components. For clarity, components of thestorage system 120/130/140 are shown in FIG. 11 as RAID 1112, 1114, and1116. In other embodiments, any of the storage devices 120/130/140 maybe used.

The storage devices 1112, 1114, and 1116 represent any of the sourcestorage devices 120, 130 and 140 shown in FIG. 1. In the illustrativeexample shown in FIG. 11, they are NAS storage devices configured asRAID 1. The storage device 1112 stores mirrored data in R_(1,1) andR_(1,2) which include the stream D₁. The storage device 1114 storesmirrored data in R_(2,1) and R_(2,2) which include the stream D₂. Thestorage device 1116 stores mirrored data in R_(3,1) and R_(3,2) whichinclude the stream D₃.

The WFD processor 1120 performs WFD on M input streams to generate Noutput streams. In the illustrative example in FIG. 11, M=3 and N=4. TheWFD processor 1120 performs WFD on the 3 input streams D₁, D₂, and D₃,and generates 4 output streams S₁, S₂, S₃, and S₄. The WFD essentiallyis the reverse operation of the WFM. To successfully recover theoriginal source stream S, at least three NAS sites should be available.This operation is a matrix multiplication of the column vector (D₁, D₂,D₃)^(T) using the following equations to recover the column vector (S₁,S₂, S₃, S₄)^(T):S ₁ =w ₁₁ ·D ₁ +w ₁₂ ·D ₂ +w ₁₃ ·D ₃  (6a)S ₂ =w ₂₁ ·D ₁ +w ₂₂ ·D ₂ +w ₂₃ ·D ₃  (6b)S ₃ =w ₃₁ ·D ₁ +w ₃₂ ·D ₂ +w ₃₃ ·D ₃  (6c)S ₄ =w ₄₁ ·D ₁ +w ₄₂ ·D ₂ +w ₄₃ ·D ₃  (6d)

The WFD processor 1120 may generate one redundant data stream S₄. Thisdata stream S₄ may be left unused or is used for integrity check againstpossible compromised stored/transported data streams.

When the M input streams are known to be generated using an envelope,the first WFD processor performs WFD on the M input streams including anenvelope to generate the N output streams including a de-envelopedoutput stream.

The de-segmenter 1130 acts as a post-processor to de-segment or to mergethe output streams S₁, S₂, S₃, and S₄ into the source stream S. Thede-segmentation is the reverse of the known segmentation in the writingor storing process.

FIG. 12 is a diagram illustrating the data transport and/or storageprocessing system 110 for receiving or reading data from two localstorage systems and two cloud storage devices according to oneembodiment. The processing system 110 is configured to correspond to theconfiguration shown in FIG. 5. The storage system 120/130/140 in FIG. 12is similar to the storage system 170/180/190 shown in FIG. 5. Thisconfiguration includes two local storage systems such as NAS devices1112 and 1114 and two cloud storage devices R₃ 1216 and R₄ 1218 via thecloud 120.

The WFD processor 1120 performs WFD on M input streams to generate Noutput streams. In the illustrative example in FIG. 12, M=3 and N=4. TheWFD processor 1120 performs WFD on the 3 input streams D₁, D₂, and D₃,and generates 4 output streams S₁, S₂, S₃, and S₄. The WFD essentiallyis the reverse operation of the WFM. As in the configuration in FIG. 11,the WFD processor 1120 may generate one redundant data stream S₄. Thisdata stream S₄ may be left unused or is used for integrity check againstpossible compromised stored/transported data streams. The de-segmenter1130 acts as a post-processor to de-segment or to merge the outputstreams S₁, S₂, S₃, and S₄ into the source stream S. The de-segmentationis the reverse of the known segmentation in the writing or storingprocess.

FIG. 13 is a diagram illustrating the data transport and/or storageprocessing system 110 for receiving or reading data from a local storagesystem and two cloud storage devices according to one embodiment. Theprocessing system 110 is configured to correspond to the configurationshown in FIG. 6. The storage system 120/130/140 in FIG. 12 is similar tothe storage system 170/180/190 shown in FIG. 6. This configurationincludes a local storage site 1310 having two storage systems such asNAS devices as RAID 1 to store data streams R₁ and R₂ in mirrored formatand two cloud storage devices R₃ 1216 and R₄ 1218 via the cloud 120.

The WFD processor 1120 performs WFD on M input streams to generate Noutput streams. In the illustrative example in FIG. 12, M=3 and N=4. TheWFD processor 1120 performs WFD on the 3 input streams D₁, D₂, and D₃,and generates 4 output streams S₁, S₂, S₃, and S₄. The WFD essentiallyis the reverse operation of the WFM. As in the configuration in FIG. 11,the WFD processor 1120 may generate one redundant data stream S₄. Thisdata stream S₄ may be left unused or is used for integrity check againstpossible compromised stored/transported data streams. The de-segmenter1130 acts as a post-processor to de-segment or to merge the outputstreams S₁, S₂, S₃, and S₄ into the source stream S. The de-segmentationis the reverse of the known segmentation in the writing or storingprocess.

FIG. 14 is a diagram illustrating the data transport and/or storageprocessing system 110 for receiving or reading data using a systematicdecoder according to one embodiment. The processing system 110 includesa WFD processor 1410 and a systematic decoder 1420. The configuration inFIG. 14 corresponds to the reverse process of the configuration in FIG.8.

The WFD processor 1120 performs WFD on M input streams to generate Noutput streams. In the illustrative example in FIG. 11, M=3 and N=2. TheWFD processor 1410 performs WFD on the 3 input streams D₁, D₂, and D₃,and generates 3 output streams S₁, S₂, and S₃. The WFD essentially isthe reverse operation of the WFM. To successfully recover the originalsource stream S, at least three NAS sites should be available. Thisoperation is a matrix multiplication of the column vector (D₁, D₂,D₃)^(T) using the following equations to recover the column vector (S₁,S₂, S₃)^(T):S ₁ =w ₁₁ ·D ₁ +w ₁₂ ·D ₂ +w ₁₃ ·D ₃  (7a)S ₂ =w ₂₁ ·D ₁ +w ₂₂ ·D ₂ +w ₂₃ ·D ₃  (7b)S ₃ =w ₃₁ ·D ₁ +w ₃₂ ·D ₂ +w ₃₃ ·D ₃  (7c)

FIG. 15 is a diagram illustrating the data transport and/or storageprocessing system 110 for receiving or reading data using a cascadedstructure for the WFD processor according to one embodiment. Theprocessing system 110 includes a WFD processor 1520, two WFD processors1512 and 1514, and a de-segmenter 1530. The processing system 110 mayinclude more or less than the above components.

The cascade structure includes two levels. In the first level, the twoWFD processors 1512 and 1514 perform WFD on the retrieved data streamsR_(1,1), R_(1,2), R_(2,1), and R_(2,2) to generate the input streams D₁,D₂, and D₃. The WFD processor 1512 operates on the two storage streamsR_(1,1) and R_(1,2) and generates 4 outputs, two of which are D₁ and D₂;the other two outputs may be unused or may be used for integrity checkagainst possible compromised stored/transported data streams. Asdiscussed above, the WFD may be performed by a matrix multiplicationusing the inverse matrix of:

$\quad\begin{bmatrix}\rho_{11} & \rho_{12} \\\rho_{21} & \rho_{22}\end{bmatrix}$

The WFD processor 1514 operates on the two storage streams R_(2,1) andR_(2,2) and generates 3 outputs, one of which is D₃; the other twooutputs may be unused or may be used for integrity check againstpossible compromised stored/transported data streams.

In the second level, the WFD processor 1520 perform WFD on the threeinput streams D₁, D₂, and D₃ to generate 3 output streams S₁, S₂, andS₃. As discussed above, the WFD may be performed as a matrixmultiplication using the inverse matrix of the matrix used to generateD₁, D₂, and D₃ in the writing or storing process.

The de-segmenter 1530 acts as a post-processor to de-segment or to mergethe output streams S₁, S₂, and S₃ into the source stream S. Thede-segmentation is the reverse of the known segmentation in the writingor storing process.

FIG. 16 is a diagram illustrating the data transport and/or storageprocessing system 110 for receiving or reading data using a cascadedstructure for the WFD processor according to one embodiment. Theprocessing system 110 in FIG. 16 is similar to the processing system 110in FIG. 15 except the number of output streams in the first level WFDprocessors. The processing system 110 includes WFD processors 1612 and1614 in the first level, the WFD processor 1520 in the second level, andthe de-segmenter 1530. The WFD processor 1612 operates on the twostreams R₁₁ and R₁₂ and generates 3 outputs, two of which are D₁ and D₂;the other output may be unused or may be used for integrity checkagainst possible compromised stored/transported data streams. The WFDprocessor 1614 operates on the two storage streams R_(2,1) and R_(2,2)and generates 2 outputs, one of which is D₃; the other output may beunused or may be used for integrity check against possible compromisedstored/transported data streams.

FIG. 17 is a diagram illustrating curves 1700 representing failure rateof a distribute storage system according to one embodiment. The curves1700 include two curves 1730 and 1740 plotted on a coordinate systemhaving a horizontal axis 1710 and a vertical axis 1720. The horizontalaxis 1710 represents the failure rate p in each storage device. Thevertical axis 1720 represents the failure rate in the system.

One can compare the storage scheme with RAID 10 in terms of the arrayfailure rate. Suppose each of the four NAS sites has a failure rate pover the next three years. If these sites are arranged in RAID 10configuration, the corresponding array failure rate over the next threeyears is α₁=1−(1−p²)⁴. If these sites are arranged in the configurationdisclosed in FIG. 2 and FIG. 11, the corresponding array failure rateover the next three years is α₂=1−4(1−p²)³p²−(1−p²)⁴. The disclosedconfiguration thus has better availability as α₁>α₂ given typical pvalues (p<½), and has better privacy protection as every NAS stores datasub-streams is identical to WFCs.

The failure rate α₁ 1730 for conventional RAID 10 configuration ishigher than the failure rate α₂ 1740 for WFMed RAID 11 configurations.At a case where individual device failure rate p at 0.4 for next 3years, the calculated failure rate α₁ for a conventional RAID 10configuration will be at 0.5 or 50% probability while the calculatedfailure rate α₂ for a WFMed RAID 11 configuration will be at 0.13 or 13%probability.

FIG. 18 is a diagram illustrating curves representing failure rate of adistribute storage system according to one embodiment.

One can compare the storage scheme with systematic code governed solelyby coder 810 (in FIG. 8) in terms of the array failure rate shown inFIG. 14. Suppose each of the four NAS sites has a failure rate p overthe next three years. If these sites are arranged in 2-plus-1 systematiccoding configuration, the corresponding array failure rate over the nextthree years is α₃=1−3p(1−p)²−(1−p)³. If these sites are arranged in theconfiguration disclosed in FIG. 8 and FIG. 14, the corresponding arrayfailure rate over the next three years isα₄=1−6p²(1−p)²−4p(1−p)³−(1−p)⁴. The disclosed configuration thus hasbetter availability as α₃>α₄ given typical p values, and has betterprivacy protection as every NAS stores data sub-stream is identical toWFCs.

The curves represent failure rates 1800 of distributed storage systemsα₃ and α₄ as functions of the failure rate of individual storage devicesor storage disks, p. The vertical axis 1820 is the failure rate in asystem, while the horizontal axis 1810 is the failure rate p in eachstorage devices. The failure rate α₃ 1830 for a systematic coder 810 (inFIG. 8) configuration for a redundancy is higher than the failure rate1840 for WFMed systematic coder configuration for 2 redundancies. At acase where individual device failure rate p at 0.4 for next 3 years, thecalculated failure rate α₃ for a conventional systematic coder 810configuration will be at 0.35 or 35% probability while the calculatedfailure rate α₄ for a WFM systematic coder configuration will be at 0.18or 18% probability.

FIG. 19 is a diagram illustrating a WF processor according to oneembodiment. The processing system 110 shown in FIG. 19 may represent theprocessing system 110, or the individual processors within theprocessing system 110. Not all of the components in FIG. 19 are presentfor a particular processor. For brevity, the following refers to theprocessing system 110, but it is noted that the architecture of theprocessing system 110 may change depending on the particular function.

The processing system 110 includes a central processing unit (CPU) or aprocessor 1910, a cache 1915, a platform controller hub (PCH) 1920, abus 1925. The PCH 1920 may include an input/output (I/O) controller1930, a memory controller 1940, a graphic display controller (GDC) 1950,and a mass storage controller 1960. The system 1900 may include more orless than the above components. In addition, a component may beintegrated into another component. As shown in FIG. 19, all thecontrollers 1930, 1940, 1950, and 1960 are integrated in the PCH 1920.The integration may be partial and/or overlapped. For example, the GDC1950 may be integrated into the CPU 1910, the I/O controller 1930 andthe memory controller 1940 may be integrated into one single controller,etc.

The CPU or processor 1910 is a programmable device that may execute aprogram or a collection of instructions to carry out a task. It may be ageneral-purpose processor, a digital signal processor, amicrocontroller, or a specially designed processor such as one designfrom Applications Specific Integrated Circuit (ASIC). It may include asingle core or multiple cores. Each core may have multi-waymulti-threading. The CPU 1910 may have simultaneous multithreadingfeature to further exploit the parallelism due to multiple threadsacross the multiple cores. In addition, the CPU 1910 may have internalcaches at multiple levels.

The cache 1915 is a first level (L1) external cache memory. It istypically implemented by fast static random access memory (RAM). Othercache levels may appear externally, such as the cache 1946. Some or allcache levels (L1, L2, and L3) may all be integrated inside the CPU 1910.

The bus 1925 may be any suitable bus connecting the CPU 1910 to otherdevices, including the PCH 1920. For example, the bus 1925 may be aDirect Media Interface (DMI).

The PCH 1920 in a highly integrated chipset that includes manyfunctionalities to provide interface to several devices such as memorydevices, input/output devices, storage devices, network devices, etc.

The I/O controller 1930 controls input devices (e.g., stylus, keyboard,and mouse, microphone, image sensor) and output devices (e.g., audiodevices, speaker, scanner, printer). It also has interface to a networkinterface card 1970 which provides interface to a network 1974 andwireless controller 1972. The network interface card (NIC) 1970transmits and receives the data packets to and from a wired, wirelessnetwork 1972 or 1974. The NIC 1970 may have one or more sockets fornetwork cables and the type of socket depends on the type of network itwill be used in. The network 1974 may be a LAN, a MAN, a WAN, anintranet, an extranet, or the Internet.

The memory controller 1940 controls memory devices such as the randomaccess memory (RAM) 1942, the read-only memory (ROM) 1944, the cachememory 1946, and the flash memory 1948. The RAM 1942 may storeinstructions or programs, loaded from a mass storage device, that, whenexecuted by the CPU 1910, cause the CPU 1910 to perform operations asdescribed above, such as WFM operations. It may also store data used inthe operations, including the input data stream or the output datastream. The ROM 1944 may include instructions, programs, constants, ordata that are maintained whether it is powered or not. This may includethe matrix coefficients used in the envelope or de-envelope process, acatalog of the envelopes, boot program, self-test programs, etc. Thecache memory 1946 may store cache data at level L2 or L3. The cachememory 1946 is typically implemented by fast static RAM to allow fastaccess from the CPU 1910. The flash memory 1948 may store programs,instructions, constants, tables, coefficients, envelopes as in the ROM1944. It may be erased and programmed as necessary.

The GDC 1950 controls the display monitor 1955 and provides graphicaloperations. It may be integrated inside the CPU 1910. It typically has agraphical user interface (GUI) to allow interactions with a user who maysend a command or activate a function.

The mass storage controller 1960 controls the mass storage devices suchas CD-ROM 1962 and hard disk 1964.

Additional devices or bus interfaces may be available forinterconnections and/or expansion. Some examples may include thePeripheral Component Interconnect Express (PCIe) bus, the UniversalSerial Bus (USB), etc.

Elements of one embodiment may be implemented by hardware, firmware,software or any combination thereof. The term hardware generally refersto an element having a physical structure such as electronic,electromagnetic, optical, electro-optical, mechanical,electro-mechanical parts, etc. A hardware implementation may includeanalog or digital circuits, devices, processors, applications specificintegrated circuits (ASICs), programmable logic devices (PLDs), fieldprogrammable gate arrays (FPGAs), or any electronic devices. The termsoftware generally refers to a logical structure, a method, a procedure,a program, a routine, a process, an algorithm, a formula, a function, anexpression, etc. The term firmware generally refers to a logicalstructure, a method, a procedure, a program, a routine, a process, analgorithm, a formula, a function, an expression, etc., that isimplemented or embodied in a hardware structure (e.g., flash memory,ROM, EROM). Examples of firmware may include microcode, writable controlstore, micro-programmed structure.

When implemented in software or firmware, the elements of an embodimentmay be the code segments to perform the necessary tasks. Thesoftware/firmware may include the actual code to carry out theoperations described in one embodiment, or code that emulates orsimulates the operations. The program or code segments may be stored ina processor or machine accessible medium. The “processor readable oraccessible medium” or “machine readable or accessible medium” mayinclude any non-transitory medium that may store information. Examplesof the processor readable or machine accessible medium that may storeinclude a storage medium, an electronic circuit, a semiconductor memorydevice, a read only memory (ROM), a flash memory, an erasableprogrammable ROM (EPROM), a floppy diskette, a compact disk (CD) ROM, anoptical disk, a hard disk, etc. The machine accessible medium may beembodied in an article of manufacture. The machine accessible medium mayinclude information or data that, when accessed by a machine, cause themachine to perform the operations or actions described above. Themachine accessible medium may also include program code, instruction orinstructions embedded therein. The program code may include machinereadable code, instruction or instructions to perform the operations oractions described above. The term “information” or “data” here refers toany type of information that is encoded for machine-readable purposes.Therefore, it may include program, code, data, file, etc.

All or part of an embodiment may be implemented by various meansdepending on applications according to particular features, functions.These means may include hardware, software, or firmware, or anycombination thereof. A hardware, software, or firmware element may haveseveral modules coupled to one another. A hardware module is coupled toanother module by mechanical, electrical, optical, electromagnetic orany physical connections. A software module is coupled to another moduleby a function, procedure, method, subprogram, or subroutine call, ajump, a link, a parameter, variable, and argument passing, a functionreturn, etc. A software module is coupled to another module to receivevariables, parameters, arguments, pointers, etc. and/or to generate orpass results, updated variables, pointers, etc. A firmware module iscoupled to another module by any combination of hardware and softwarecoupling methods above. A hardware, software, or firmware module may becoupled to any one of another hardware, software, or firmware module. Amodule may also be a software driver or interface to interact with theoperating system running on the platform. A module may also be ahardware driver to configure, set up, initialize, send and receive datato and from a hardware device. An apparatus may include any combinationof hardware, software, and firmware modules.

It will be appreciated that various of the above-disclosed and otherfeatures and functions, or alternatives thereof, may be desirablycombined into many other different systems or applications. Variouspresently unforeseen or unanticipated alternatives, modifications,variations, or improvements therein may be subsequently made by thoseskilled in the art which are also intended to be encompassed by thefollowing claims.

What is claimed is:
 1. An apparatus comprising: a pre-processor tosegment or to code a source stream to produce M input streams; a firstwavefront multiplexing (WFM) processor to perform WFM on the M inputstreams to generate N output streams; and a second WFM processor toperform WFM on the N output streams to produce storage streams to bestored in at least one of a plurality of storage devices.
 2. Theapparatus of claim 1 wherein the plurality of storage devices includesat least one of a network attached storage (NAS) device, a direct accessstorage (DAS) device, a storage area network (SAN) device, a redundantarray of independent disks (RAIDs), a cloud storage, a hard disk, asolid-state memory device, and a device capable of storing data.
 3. Theapparatus of claim 1 wherein the first WFM processor performs WFM on theM input streams including an envelope to generate the N output streamsincluding an enveloped output stream which is substantially identical tothe envelope.
 4. The apparatus of claim 1 wherein the pre-processorcomprises a systematic coder to encode the source stream with asystematic code.
 5. The apparatus of claim 4 wherein the systematic codeincludes an error-correcting code.
 6. An apparatus comprising: a firstwavefront demultiplexing (WFD) processor to perform WFD on storagestreams retrieved from at least one of a plurality of storage devices togenerate M input streams; a second WFD processor to perform WFD on the Minput streams to produce N output streams; and a post-processor tode-segment or to decode the N output streams into a source stream. 7.The apparatus of claim 6 wherein the plurality of storage devicesincludes at least one of a network attached storage (NAS) device, adirect access storage (DAS) device, a storage area network (SAN) device,a redundant array of independent disks (RAIDs), a cloud storage, a harddisk, a solid-state memory device, and a device capable of storing data.8. The apparatus of claim 6 wherein the first WFD processor performs WFDon the storage streams to generate the M input streams including anintegrity check stream.
 9. The apparatus of claim 6 wherein thepost-processor comprises a systematic decoder to decode the N outputstreams with a systematic code.
 10. The apparatus of claim 9 wherein thesystematic code includes an error-correcting code.
 11. A methodcomprising: performing a pre-processing operation to segment or to codea source stream to produce M input streams; performing a first wavefrontmultiplexing (WFM) operation on the M input streams to generate N outputstreams; and performing a second WFM operation on the N output streamsto produce storage streams to be stored in at least one of a pluralityof storage devices.
 12. The method of claim 11 wherein the plurality ofstorage devices includes at least one of a network attached storage(NAS) device, a direct access storage (DAS) device, a storage areanetwork (SAN) device, a redundant array of independent disks (RAIDs), acloud storage, a hard disk, a solid-state memory device, and a devicecapable of storing data.
 13. The method of claim 11 wherein the M inputstreams include an envelope and performing the WFM on the M inputstreams comprises generating the N output streams including an envelopedoutput stream which is substantially identical to the envelope.
 14. Amethod comprising: performing a first wavefront demultiplexing (WFD)operation on storage streams retrieved from at least one of a pluralityof storage devices to generate M input streams; performing a second WFDoperation on the M input streams to produce N output streams; andperforming a post-processing operation to de-segment or to decode the Noutput streams into a source stream.
 15. The method of claim 14 whereinthe plurality of storage devices includes at least one of a networkattached storage (NAS) device, a direct access storage (DAS) device, astorage area network (SAN) device, a redundant array of independentdisks (RAIDs), a cloud storage, a hard disk, a solid-state memorydevice, and a device capable of storing data.
 16. The method of claim 14wherein performing the WFD operation on the storage streams comprisesgenerating the M input streams including an integrity check stream.